Skip to content

Added compression and switched to recursive binary serialisation for asset packing#69

Merged
jonjondev merged 11 commits intomainfrom
feature/asset-compression
Aug 5, 2025
Merged

Added compression and switched to recursive binary serialisation for asset packing#69
jonjondev merged 11 commits intomainfrom
feature/asset-compression

Conversation

@jonjondev
Copy link
Copy Markdown
Member

Description

This PR updates the packing system to perform compression on packed data via zlib where the resource system is equipped to handle decompression of loaded pack file. This feature functions in concert with a newly added binary serialisation method for asset data using recursive template functions. The results show the renderapp pack file size reduced in on-disk size from 6.4MB down to 2.8MB.

The recursive templates for binary serialisation work as so at the high level, allowing any type to recursively request the serialisation of data (likely members) for any type that provides a template function to do so as below:

struct BaseVertex
{
    Vec3 position;
    FColour color;
    Vec3 normal;
    Vec2 uv;
};

struct StaticMeshData
{
    std::vector<uint32_t> indices;
    std::vector<BaseVertex> vertices;
};

inline void serialise(BinarySerialisation::Buffer& buffer, BaseVertex& value, BinarySerialisation::SerialisationMode mode)
{
    serialise(buffer, value.position, mode);
    serialise(buffer, value.color, mode);
    serialise(buffer, value.normal, mode);
    serialise(buffer, value.uv, mode);
}

inline void serialise(Buffer& buffer, StaticMeshData& value, SerialisationMode mode)
{
    serialise(buffer, value.indices, mode);
    serialise(buffer, value.vertices, mode);
}

The functionality then allows for more common and fundamental types to receive serialisation based on native serialisation rules as defined in utils/BinarySerialisation.h, such as STL containers, primitive C++ types, and core Siege types.

In addition to these new features, the following changes have also been made:

  • pack-assets targets in all Makefiles have been updated to perform their deletions before running
  • Added and updated unit testing for resources and binary serialisation

The PR has been...

  • provided a reasonable name that is not just the branch name (e.g "Added Vulkan render delegate")
  • linked to its related issue
  • assigned a reviewer from the team
  • labelled appropriately

The code has been...

  • made mergable and free of conflicts in relation to master (according to GitHub)
  • tested in a packaged state using the package targets
  • pulled to the reviewer's machine and reasonably tested

@jonjondev jonjondev requested a review from Raelr August 4, 2025 12:16
@jonjondev jonjondev added build Relating to the engine's build system utils Relating to the engine's utils labels Aug 4, 2025
Copy link
Copy Markdown
Contributor

@Raelr Raelr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awful. I hate everything that I see here and you should be especially ashamed of the use of smart pointers.
LGTM 👍

Comment thread engine/utils/BinarySerialisation.h
Comment thread packer/src/main.cpp Outdated
Comment thread engine/resources/PackFile.cpp Outdated
@jonjondev jonjondev merged commit 847a93f into main Aug 5, 2025
4 checks passed
@jonjondev jonjondev deleted the feature/asset-compression branch August 5, 2025 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Relating to the engine's build system utils Relating to the engine's utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants